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Abstract 

Objective. The main objective of this paper is to investigate whether incorporating an electronic optional guideline tool 
(EOGT) in the standardized referral template used by general practitioners (GPs) when referring patients to specialized 
care can improve outpatient referral appropriateness. Design. Intervention study with an intervention and a control group. 
Setting. 210 GPs in the municipality of Bergen and the Department of Thoracic Medicine at Haukeland University 
Hospital. Subjects. 2400 patients referred to the Department of Thoracic Medicine at Haukeland University Hospital. 
Results. An electronic optional guideline tool (EOGT) was implemented on 93 of 210 GPs' computer systems. The 
referral quality and the time spent reviewing each referral were evaluated by the hospital specialists. The GPs did not 
know that their referrals were being evaluated. The specialists were blinded with regard to information concerning the 
intervention and the control group. The specialists reported significantly higher referral quality and considerably less 
time spent on evaluating referrals when using the EOGT, with an overall time reduction of 34%. Likewise, GPs also 
reported that the EOGT was easy to use, time-saving and led to an improved quality of their referrals. Conclusion. This 
study documents an improvement in the quality of the referrals. Since the GPs save time by using the EOGT, there is 
no reason to believe that they will discontinue using it. In fact, the tool may be even more beneficial for the GP. The 
authors believe that it is possible to implement the EOGT as a standard referral tool within various fields of medicine 
and are currently in the process of developing these tools. 
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Introduction 

Every year Norwegian GPs spend 150 work years 
writing approximately 1.9 million referrals to special- 
ists [1]. The quality of these referrals is important 
both for the appropriate care of the patients and 
for the economics of the health-care system [2]. 
However, the number of work years consumed 
by specialists in processing these referrals is not 
known. 

Recent studies have shown that information com- 
municated through referrals is often insufficient in 
spite of their far-reaching implications and the impor- 
tance of their being appropriate [3,4] .This means that 



the process of managing referrals between GPs and 
specialists is in need of improvement to secure the 
interests of the patients and relieve specialists of the 
extra time spent on managing poor referrals [5,6]. 
Previous intervention studies on the quality of GP 
referrals have shown improvements [7,8] but have 
indicated that it is difficult to maintain the results 
over time [7,8] . The aim of this study was to investi- 
gate whether incorporating an electronic optional 
guideline tool (EOGT) in the standardized referral 
template used by GPs when referring patients to spe- 
cialized care could improve outpatient referral appro- 
priateness and maintain this improvement over time. 
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• The specialists reported significantly higher 
referral quality and considerably less time 
spent on evaluating referrals written by GPs 
using the electronic optional guideline tool 
(EOGT). 

• The overall time reduction using the EOGT 
was 34%. 

• The GPs reported that the EOGT was easy 
to use, time-saving, and led to improved 
quality of their referrals. 



Materials and method 

In this study one GP and three pulmonary special- 
ists at the Department of Thoracic Medicine at 
Haukeland University Hospital have collaborated to 
improve the quality of referrals from GPs to the spe- 
cialists at the hospital. Collectively they have com- 
piled a set of indicators for three different diagnoses, 
namely sleep apnoea (SA), chronic obstructive pul- 
monary disease (COPD), and tumours of the lung 
(LT). These indicators represent information that 
the pulmonary specialists consider to be most impor- 
tant regarding the review of referrals from GPs. The 
GPs also considered that using these indicators was 
feasible in general practice. Based on these indica- 
tors, the EOGT was created in order to increase the 
effectiveness and efficiency of the referral process. 

The outpatient clinic at the Department of Tho- 
racic Medicine, Haukeland University Hospital, 
annually receives about 2400 patient referrals. Of 
these, 60% are diagnosed with SA, COPD, or UT. 
For each of the diagnoses seven or eight indicators 



were listed. When the GPs wrote a referral to a spe- 
cialist using this tool, the GPs first decided on a ten- 
tative diagnosis using clinical skills and knowledge. 
If this tentative diagnosis was an UT, SA, or COPD, 
the EOGT provided an information box (Figure 1) 
corresponding to the diagnosis at hand. This pro- 
vided a reminder regarding which information the 
GP should provide in order to increase the appropri- 
ateness of the referral. In Norway it is common prac- 
tice for most GPs to write the referral while the 
patient is present in the consultation room. This 
means that the GP can use the EOGT guidelines to 
gather information about the patient directly and 
simultaneously write the referral. 

The EOGT was implemented in Infodoc Plena- 
rio, a computer system currently used by 93 GPs in 
the municipality of Bergen (n = 93). This took place 
in conjunction with a new version of the software 
being installed. This version included many improve- 
ments in addition to the EOGT. The remainder of 
the GPs (n = 117) who were not using this particu- 
lar computer system were included in a control 
group. The GPs were not informed that their refer- 
rals would be subject to anonymous evaluation 
and comparison with a control group in this study. 
During the intervention period the same evaluation 
form was used to evaluate all referrals sent from 
these GPs to the Department of Thoracic Medicine, 
Haukeland University Hospital. This was performed 
over a period of nine months by six specialists in 
pulmonary medicine. They were all consultants 
working full time. The specialists were blinded from 
all information regarding the intervention and con- 
trol groups. The amount of time the specialist used 
to consider and manage each referral was recorded 
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Figure 1. The information box above appeared in the upper right corner of the electronic standardized referral template when the 
GP selected COPD, SA, or LT as the reason for referral. 
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using a stopwatch. They were instructed to record 
30 to 60 seconds as one minute and less than 30 
seconds as zero. 

The evaluation forms contained two identical 
questions regarding referrals of all three diagnoses, 
namely: "Does the referral communicate a clear 
message?" and "Is the referral overall easily read?" 
In addition, there were seven other questions regard- 
ing referrals for SA and eight questions regarding 
LT and COPD. The specialists were instructed to 
score the referrals/diagnoses on a scale from zero to 
12, 12 being the optimal score. The total score for 
each referral was calculated thereafter. 

After the evaluation the GP intervention group 
was also interviewed by telephone regarding whether 
they used the new tool (yes/no), whether they 
found it easy to use (yes/no/indifferent) and whether 
their time expenditure on writing referrals after 
the electronic tool was implemented had changed 
(increased/decreased/unchanged). The GPs were 
also asked whether they felt that their referrals were 
more appropriate, the same, or less appropriate than 
before using the new tool. 

Statistical method 

Descriptive statistical methods were used to analyse 
the data collected during this study. Tests between 
the two continuous variables "time" and "total 
score" were carried out using the Wilcoxon test. The 
chi-square or Fisher's exact test was applied when 
comparing categorical variables. The significance 
level was 0.05. To account for multiple effects, the 
significance level was adjusted using the Bonferroni 
correction, thus leading to an adjusted level of 
0.0071 for SA (seven tests) and 0.00625 for COPD 
and LT (eight tests). 



Results 

Every GP in the municipality of Bergen was included 
in this study (n = 210). Of these, 93 constituted the 
intervention group. There were no differences 



between the two groups with regard to main vari- 
ables such as gender, age, and whether or not they 
were specialists in general practice. During the nine 
months in which the study took place, a total of 
1080 new referrals for the three diagnostic groups 
SA, COPD, and LT were received by the hospital. 
Of these, 664 referrals (70%) were from GPs. A 
small number of referrals may have come from 
GPs in districts not included in the study. These, 
however, do not use the Infodoc Plenario system 
and can therefore be considered as part of the 
control group. 

The evaluation scores in this study show an aver- 
age overall improvement of almost 30% for all three 
diagnostic groups. The two questions in common 
for evaluating all referrals showed significant differ- 
ences between the intervention and control group 
(Figure 2). 

Two of the parameters regarding lung tumours 
showed no significant differences, namely chest X-ray 
(p = 0.142) and CT thorax (p = 0.234) (Figure 3). 
All other parameters showed significant differences 
(Figure 3). 

There were no signs of burnout tendencies on the 
degree of intervention effect when comparing the 
results from the first two with the last two months 
when analysing the two questions in common for 
evaluating all referrals (see Figure 2). These two 
groups were used to compare the degree of burnout 
since these groups were large enough to be statistically 
significant. All (the) questions with the exception of 
two showed this tendency; however, some numbers 
were too small to use in the statistical calculation. The 
average total amount of time spent by the specialists 
on evaluating referrals from GPs in the intervention 
group was 34% shorter than in the control group (SA 
42%, COPD 23%, and LT 41%); 82 of the GPs had 
used the EOGT The 1 1 GPs not using the tool 
reported that they had not been aware that the tool 
was implemented in the new version of their computer 
software. Of the 82 GPs who reported using the tool, 
96% found it saved time, and all found it easy to 
use and were under the impression that they recorded 
better information for the specialists to review. 
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Figure 2. This figure shows the two main questions in common for all three diagnoses and that the use of these two questions results 
in a significant improvement in the most important part of a referral: Does the referral communicate a clear message and is the referral 
easily read. 
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Figure 3. Percentage of the answers to questions 3-10 for sleep apnoea, chronic obstructive pulmonary disease, and lung tumours in 
the intervention and the non-intervention group in numbers and percentages with p-values of the chi-squared test (*) or Fisher's exact 
test (**). 



Discussion 

One weakness of the study could be that the GPs in 
Bergen were not randomized into an intervention 
and a control group, making system bias a possible 
weakness. The GPs using Infodoc Plenario received 
the EOGT as part of a new version of the software 
containing other additional improvements. One could 
argue that the GPs who had installed the newest ver- 
sion of the Infodoc software system are more atten- 
tive and may have written better referrals with or 
without the EOGT. The analysis of the intervention 
and control group showed no differences in whether 
the GP was a specialist or not, which strengthens 
our belief that this systematic bias does not occur. 
Furthermore, neither the intervention group nor the 
control group knew that their referrals were subject 
to anonymous evaluation; 88% of the GPs in the 
intervention group used the EOGT at all times. 
The referrals made by 12% of the GPs will not cause 
any methodological problem because it will only 
reduce the differences between the two groups. 

We believe one weakness of the study is that 
some referrals were included from GPs outside 
Bergen. However, none of these was written using 
the EOGT as these GPs do not use the Infodoc 
Plenario software. This may explain why 44% of 
the GPs in the intervention group wrote only 33% 
of the referrals. We also have to take into consider- 
ation that 1 1 of the GPs included in the interven- 
tion group did not use the EOGT. 

The GPs were interviewed after the evaluation 
was completed; however, they were still using the 
EOGT. At the time of the interview the GPs had 
become more comfortable using the tool and this 
may have positively influenced the answers given in 
the interview. 

A meta-analysis showed that interventions 
involving advice obtained from a structural referral 
sheet were the only effective measure for improving 



a referral [9]. Other studies have shown the same 
results [6,7] . Our study supports this flnding, as only 
two parameters were unchanged, chest X-ray and 
CT scan of lung tumours, which not unexpectedly 
had a high score both in the control and in the 
intervention group. Several intervention studies show 
that the effect of an intervention diminishes over 
time if the intervention is not repeated regularly 
[7,8]. The greatest beneflt of the EOGT is that it 
serves as an ongoing intervention because it appears 
every time the GP uses the structured referral tem- 
plate. Thus, its effect is more likely to be withheld. 
Comparing the first with the last months of the study 
we found no tendency towards burnout. 

Grimshaw et al. pointed out that referral guide- 
lines are more likely to be effective if local secondary 
care providers are involved with GPs in making guide- 
lines [9] . It is important to both reflect local circum- 
stances and address local barriers. This survey was a 
collaborative project between well-respected, local 
pulmonary specialists at the hospital and an experi- 
enced GP. This may also contribute to explaining 
the great improvement in the intervention group. 

It has been shown that the main disadvantage of 
structural referral templates is that more time is 
required to fill them out and that their effect dimin- 
ishes with time [7,9]. To meet this challenge, this 
study has attempted to provide the GPs with a tool 
that is easy to use and contains structural advice 
from specialists, which reduces the workload, saves 
the GPs time, and helps them write referrals of bet- 
ter quality. The scores in this study show an average 
overall improvement of almost 30% for all three diag- 
nosis groups. This corresponds with another study 
that assesses quality improvement of referrals from 
primary health care [10]. Ranvolt et al. found a 36%) 
increase in adequate referrals using an intervention 
composed of simple but precise guidelines for refer- 
rals in paper format. Their results were based on a 
registration period of only three months compared 
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with our study that lasted nine months [10]. The 
Danish study also showed that more than 70% of 
referrals to gynaecologic and orthopaedic depart- 
ments did not contain any information on the 
patients' use of medicines [10]. Our study showed 
that no information on the patients' use of medicines 
in the control group was provided in 44% of the 
referrals for sleep apnoea and 58% for referrals 
regarding COPD (see Figure 3). Using the optional 
guidelines there was a significant increase in both 
groups (SA and COPD) after the intervention of 
44%) to 76%) and 58% to 87%, respectively. 

Kristoffersen et al. show that in order to succeed 
in implementing an EOGT, the tool has to function 
well in day-to-day general practice and be easy to 
use [12]. Heimly tried to implement an inter- 
active EOGT, but concluded that this was too 
time-consuming for the GPs [13]. 

The negative outcome of implementing referral 
guidelines in general practice developed by hospital 
specialists alone has been discussed by Jiwa et al. 
[11]. They pointed out that specialists were too 
demanding and that it was too time-consuming 
for the GPs to collaborate. In our study we have 
shown that not only the specialists but also the GPs 
saved time using the guidelines. After using the 
EOGT for one year the GPs in Bergen forwarded 
a request that Haukeland University Hospital take 
charge of developing EOGTs for other fields of med- 
icine. This is currently taking place. 

Conclusion 

Time is limited in outpatient clinics today and the 
demand for efficiency is therefore high. Hence, new 
methods are necessary to improve efficiency and 
quality at the outpatient clinics. Our study docu- 
ments an improvement in the quality of the referrals. 
Since the GPs save time using the EOGT and report 
that they find it very useful, there is no reason 
to believe that they will discontinue using it. The 
EOGT benefits all parties, in particular the GP. 
Therefore it is logical to assume that the interven- 
tion tool will continue to have a time-saving and 
quality-improving effect on the referrals. We believe 
that it is also possible to implement the EOGT as a 
standard referral tool within various other fields of 
medicine on a national basis for all GP software sys- 
tems. Haukeland University Hospital is currently 
developing such tools for investigating urological 
and gastroenterological cancer diagnoses. 
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